Not applicable.
Not applicable.
Not applicable.
This invention relates to artificial intelligence systems and more particularly to the organization and structure of a plurality of learning artificial intelligence entities.
For purposes of this document, we consider an artificially intelligent (AI) entity as having three defining properties. Two are conventional within the AI discipline; the third is sometimes used and sometimes omitted, depending on the emphasis of the AI effort.
First, an AI entity exhibits complex behavior that affects the world external to itself. It may send control information to electronic or mechanical devices; it may output information to human beings; it may directly alter some property of its environment. Second, an AI entity responds to information about its environment. Its xe2x80x98sensesxe2x80x99 may be electronic readings, digitally coded information, physical movement or any other method of bringing information from outside. In general usage, xe2x80x98complexxe2x80x99 behavior means xe2x80x98non-obviousxe2x80x99 behavior. For example, a simple controller like the governor on a steam engine would not usually be considered artificially intelligent since the source of its response to sensed engine speed is apparent to observation.
AI devices with these two properties exhibit complex behavior in an unchanging way. Examples in widespread current use would be (1) xe2x80x98expert systemsxe2x80x99, where a set of facts and rules is input to an execution device which will then, in the absence of new inputs, give the same answers to the same questions, (2) stock charting systems, where the rules for choosing investments, once defined, make the same recommendations whenever the same patterns appear, and (3) xe2x80x98multi-agent systems,xe2x80x99 AI applications in resource allocation where the xe2x80x98agentsxe2x80x99 are executing fixed algorithms and are given a language or protocol in which to communicate and negotiate with each other.
The third property in the present definition is that the AI entity changes its behavior as a result of experience. That is, the same situation will evoke a different response from the AI entity if the entity has xe2x80x98seen itxe2x80x99 before. We say that such an entity is a xe2x80x98learning AI entityxe2x80x99.
To summarize, an AI entity accepts sense data from its environment, produces complex behavior in response, and as the definition is used here learns from experience.
Current AI in the non-learning sense includes knowledge bases and multi-agent processing schemes. Knowledge bases are organized around collections of information with rules for making inferences and answering queries. Multi-agent schemes combine numerous entities operating on fixed algorithms. Often these aggregations include convenient methods for people to update the algorithms, inference rules and other recipes that govern their behavior. However, the xe2x80x98learningxe2x80x99 is actually happening in their human keepers, but not on the aggregation itself.
Current AI learning technology consists largely of refinements of two basic models developed in the 1960s, as described in the next section.
The Bases of Computer Artificial Intelligence
Single Entity and Scoring Polynomial (Newell, Samuel)
The 1958 paper by Newell, Shaw and Simoni and the 1959 paper by Samuelii laid the groundwork for the single AI entity using the scoring polynomial approach. In Newell, et al., a chess-playing automaton is described. Samuel""s version played checkers. In both cases the xe2x80x98sensesxe2x80x99 consisted of various measures of game positions. In chess, measures like point values of pieces for each side, occupancy of key center squares, control of long files, etc., were used. A move generator created a list of possible chains of moves and countermoves, ending in a list of accessible future positions. Each position had its sense values, and the imputed value of each position was the sum of each sense value multiplied by a factor specific to that sense. Learning, a major factor in the Samuel paper, involved adjusting the factors applied to each sense by applying feedback from positions actually attained.
i Newell, A., J. C. Shaw, and H. A. Simon. 1958. Chess-Playing Programs and the Problem of Complexity. IBM J. Res. Develop. 2:320-25. 
ii Samuel, A. L. 1959. Some Studies in Machine Learning Using the Game of Checkers. IBM J. Res. Develop. Pp. 210-229. 
The defining characteristics of this model, then, are (1 ) the single entity using a defined set of senses and a scoring polynomial, and (2) reinforcement by adjustment of the sense factors in the polynomial.
Neural Net (Rosenblatt)
The Rosenblattiii model, named the Perceptron, attempted to mimic the action of neurons in animals. It was used in a simple character-recognition activity. A large number of identical cell-like entities, each exhibiting simple behavior, were connected, each to all others. Senses were applied to some cells, which propagated simple on-off pulses to other connected cells. Reinforcement was applied to other cells, which also sent on-off pulses to their connected neighbor cells. Cells receiving pulses would transmit pulses to their own connected neighbors if their total receipts exceeded a threshold value unique to that cell. Learning consisted of adjusting the individual cells"" thresholds based on reinforcement pulses received.
iii Rosenblatt, F. 1958. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Cornell Aeronautical Laboratory, Psychological Review, v.65, No. 6, p. 386-408. 
The defining characteristics of the Rosenblatt model, then, are (1) a large number of simple threshold-type cells working with on-off pulses, (2) initial connection of cells to neighbors, and (3) learning by adjustment of thresholds.
Current art encompasses the Newell/Samuel models of single AI entities, which are able to sense environmental input, exhibit complex behavior, and learn through use of various scoring methods. The single-entity scoring polynomial is used in such areas as scoring of loan applications, although in practice the learning process is xe2x80x98frozenxe2x80x99 to prevent unpredictable behavior in a business environment. There is also a great deal of current art based on the Rosenblatt neural net model. Neural net models based on the original Perceptron actually learn in operation in, for example, stock-picking applications. While they have grown in complexity by xe2x80x98layeringxe2x80x99, connecting multiple xe2x80x98simplexe2x80x99 Rosenblatt assemblages, they are still based on the relay-line threshold-activated undifferentiated cell.
There have been no combinations of the single complex learning (Newell) entity into complex assemblages including role differentiation and internally driven learning. However, such an AI super-entity constructed of an arrangement of modular learning AI entities, role differentiated and hierarchically organized, and motivated by policies set for subordinates by their superiors, would more accurately model such super-intelligent entities as communities, teams, societies, or corporations.
Accordingly, there is a need in the art for a form of AI entity that combines the cooperative aspects of the simple Rosenblatt model with the more sophisticated individual behavior of the Newell-Samuel model, adding to standard modular form the new elements of role differentiation and variation of behavior as a result of experiencexe2x80x94both the direct experience of the entity and that of other entities.
Further, there is a need in the art for a mode of integration of AI entities of this type with other entities, including human beings, in a cooperative network using the same communication structures interchangeably.
Further, there is a need in the art for the learning behavior of the super-entity created by linking numerous AI entities, and the application of this super-entity to complex problems and to simulation of actual multi-entity situations.
The invention is an artificial intelligence entity incorporating a structure not seen in prior art. Specifically, the AI learning entity is modular, so that a single entity is replicated many times to form a super-entity that shows intelligent behavior transcending that of its individual constituents. We refer to the modular AI learning entity as a golemiv (20). It is role differentiated, in that structurally identical entities perform different functions and exhibit different behavior depending on their personasv and the learning they have completed as driven by other entities. Further, the group of golems is hierarchically organized, in the sense that xe2x80x98superiorxe2x80x99 entities issue policies to xe2x80x98subordinatexe2x80x99 entities. The golem responds to xe2x80x98sensexe2x80x99 input from its environment as well as to policy requirements set by other entities.
iv Golem: In Jewish legend, a human being made of clay and given life by supernatural means. Hence, a robot or automaton. 
v Persona: The mask worn by a player in ancient Greek comedy and drama. Hence, the set of characteristics associated with a role. 
The hierarchical organization of golems in this invention differs from other hierarchical organization schemes. In some such schemes the hierarchically organized entities are not learning entities but obtain changes to their evaluation mechanisms from human input. In other cases, the learning mechanism is artificially restricted and lacks the golem-teach-golem reinforcement mechanism of the present invention. An example of the latter is U.S. Pat. No. 5,367,449 to Manthey on Nov. 22, 1994. In the Manthey patent, a single artificial intelligence system employed a hierarchical scheme of identical AI entities working against discontinuous external inputs (ie, inputs limited to a fixed set of values rather than the continuous variables in the present invention). Further, the inputs were required to be independent and uncorrelated, a requirement not part of the present invention and difficult to meet in many real situations. No variation in persona (i.e., entity capabilities or role differentiation) was included. In contrast, the artificial intelligence entity described here incorporates hierarchical organization of a plurality of golems differentiated in role and potentially in type (i.e., including humans and other AI entities) within a super-entity.
We use several terms to describe how the golems, through differences in persona and hierarchical arrangement, derive individualized behavior despite underlying structural sameness. The xe2x80x9crolexe2x80x9d of a golem is defined by the collection of policies and action types available to it; thus two golems may have identical roles, or may be role differentiated by different policy sets or available action types.
We define a golem""s persona more broadly, as the list of sense statements, actions, and policies it can understand and a corresponding set of weights for turning these lists into rankings of actions which it might choose to take. Thus two golems who share a role can have either identical, or different, personas. We can characterize a golem""s persona as its individualized representation of the role it may share with others. Further, it is through changes to its persona, both self-initiated and initiated by actions of a golem""s superior(s), that a golem implements learning.
In this model the golem can perform actions under its own controlxe2x80x94either direct actions upon its environment or policy actions to its subordinate entities. In contrast with non-learning artificial intelligence, each golem independently learns by using success-failure information, defined in terms of the policies in effect, to modify its future behavior, specifically by modifying its evaluation of alternative actions. Each golem is also presented with a random influx of new, untried sense statements and policies for its use in evaluating and learning. In this hierarchical model, a golem""s success is measured in terms of policies set by its superior, so that overall there is a policy reinforcement loop among entities and role differentiation is supported.
The golem which is the subject of this invention offers an effective method of multiplying the learning capability of simple AI entities through hierarchical organization and reinforcement. It also allows decentralization of an AI process without loss of linked learning capability. This is particularly useful given the current growth in feasibility of networked information structures. Hence, the golem is a useful artificial intelligence tool and thus brings added utility to any context where artificial intelligence is currently applied. Additionally, the golem has significant potential for use as a modeling tool; for example, an AI super-entity constructed of an arrangement of golems, role differentiated and hierarchically organized, and motivated by policies set for subordinates by their superiors, more accurately models real-world super-intelligent entities (e.g., communities, teams, societies, or corporations).
The golem is novel in the current and prior art in that it offers a mode of learning and reinforcement in hierarchical structures without constraints on externally derived inputs (senses) such as that they be mutually exclusive or limited to discontinuous values. It also offers a novel method of reinforcement of AI entities by other AI entities using its hierarchical scheme.
It is helpful to have a concrete example in explaining the invention. The following discussion is directed to a computer apparatus that is able to accept computer-readable data input, store computer-readable data, manipulate computer-readable data, and communicate computer-readable data output; in short, a computer platform onto which the scheme of golems can be encoded.
Modular AI Entity
The invention consists of a modular AI learning entity, which we refer to as a golem (20). A single golem is replicated many times to form a super-entity that shows intelligent behavior transcending that of its individual constituents. Within the group of golems, individual golems occupy roles. One golem may xe2x80x98commandxe2x80x99 several other entities. Not all roles need to be occupied by the golems described here; roles can also be taken by other kinds of AI entities or by human beings, using an interface (such interface fulfilling the function whereby each of said foreign artificially-intelligent entities and human beings can interface with the modular artificial intelligence learning entities).
Hierarchically Organized
The group of entities is hierarchically organized, in the sense that xe2x80x98superiorxe2x80x99 entities issue policies to xe2x80x98subordinatexe2x80x99 entities. However, the hierarchy need not be a simple xe2x80x98treexe2x80x99 hierarchy; more complex arrangements are possible.
Golem Responds to External xe2x80x9cSensexe2x80x9d
Like all AI entities, the golem described here responds to external senses. An example: The golem occupies the role of second baseman in a baseball game. Sense data is: There are men on first and third, the ball is hit to me
Golem Responds to xe2x80x9cPolicy xe2x80x9d Inputs from Other AI Entities
In addition to sense data from the external world, the golem described here responds to policy requirements set by superior entities. In the baseball example, the second baseman""s superior entity (manager) could have said xe2x80x98Choke off runxe2x80x99 or xe2x80x98Try for the double playxe2x80x99. Which policy was in effect would partially determine the second baseman""s action.
Golem Performs Actions
Actions taken by a golem can be either xe2x80x9cdirect actions,xe2x80x9d which have an effect on the golem""s persona or on the external environment, or xe2x80x9cpolicy actions,xe2x80x9d directed toward the golem""s subordinates.
Golem Performs Direct Actions
In this model the golem can perform actions under its own control. It does this either directly or by issuing commands to a non-intelligent device. In the baseball example, the second baseman has some action options: Throw to home, throw to first, throw to third, throw to home, do nothing. The results of direct actions are reflected in the environment, where they can be sensed.
Golem Performs Policy Actions
The golem may also perform policy actions, either by issuing policies to its subordinate entities if it has any, or by directing the reinforcement of successful decision making by its subordinates.
The policies issued by the golem to its subordinates would be determined by the senses available to the issuing golem. In the baseball example the second baseman has no subordinates. The manager has subordinates. Prior to the pitch, the manager might issue xe2x80x98choke off the runxe2x80x99 (say, the team trails by one run in the bottom of the ninth inning). Alternatively, the manager might issue xe2x80x98go for the double playxe2x80x99 (say, the team leads by three in the top of the fifth).
Golem Learns from Success and Failure
The golem performs its own actions and issues policy orders to subordinates in keeping with its own policy orders (received from a superior) and its sense impressions. The intent of these actions is to execute those policies successfully. In the baseball example, the second baseman""s action under the xe2x80x98choke off the runxe2x80x99 policy is successful if no run scores. Under xe2x80x98get the double playxe2x80x99 it is successful if the double play comes off.
Learning, for the golem, then consists of using success-failure information, defined in terms of the policies in effect, to modify the golem""s future behavior. It does this by modifying the golem""s evaluation of alternative actions.
Golem is Role Differentiated
The golem""s role consists of its full set of policies and action types, which it shares with all other golems fulfilling the same role. Golems with access to differing policies or action types are thereby role-differentiated. A golem, moreover, executes its role by considering the sense statements available to it and evaluating which actions to take through use of its own set of weights. This combination of its role together with its defined sense statements and set of weights constitutes the golem""s persona, and it is the persona that allows the golem to act differently than may other golems in the same role. Thus the super-entity, through the hierarchically organized golems, supports both role differentiation and individualized behavior within roles.
Further objects and advantages of the invention will become apparent from a consideration of the drawings and ensuing description.